Complex computer software is typically developed incrementally. That is, as functionality is added, and as bugs are detected and fixed, different “builds” of the software are created. With each build (which represents an intermediate version of the software), the software is tested, enhanced, and refined. Through a series of builds, the software is incrementally improved until a final, fully operational version of the software is derived. For a highly complex set of software, there can be hundreds or even thousands of builds before the final version of the software is derived.
As noted above, each build is tested to ensure that it is working properly. This usually involves the running of a series of tests on each build. Even if a test passes on one build, that test is still run on subsequent builds to make sure that whatever was done in the most recent build has not broken any functionality that was working properly in previous builds. As a result, in many cases, the same tests are run over and over again on different builds.
The testing of builds is often done manually. That is, a build is often tested by having a user execute the software on a computer, and then having the user select certain functionality through a series of user interface actions (e.g. mouse clicks, keyboard actions, etc.) on certain user interface components (e.g. buttons, menus, etc.). The user then observes the results of the selections to determine whether the software operated properly. Testing a build manually is fine if the number of builds is relatively small. However, as noted above, there can be hundreds or even thousands of builds for a complex set of software. Having a user manually test this many builds is slow, tedious, costly, and generally wasteful.
To reduce the amount of manual effort required, some testers have written testing programs. A testing program basically takes the place of the user by programmatically initiating the user interface actions (e.g. keystrokes, mouse clicks, etc.) on the software that the user otherwise would have had to initiate manually. With a testing program, the amount of user involvement can be significantly reduced. However, the testing program approach has some significant drawbacks. One drawback is that it requires a user with sufficient programming skills to write the program. Many testers do not have such skill. Another drawback is that it requires the tester to have intimate knowledge of the internals of the software to be tested. Often, the tester is not the person who developed the software, and hence, does not have such familiarity with the software. Furthermore, the testing program may have to be changed to accommodate changes in the software, which means that the testing program may have to be constantly updated. In some cases, the amount of time and effort required to write and to update the testing program may be almost as great as testing the various builds manually. In view of these shortcomings, an improved methodology is needed.